fixes to make mutmut3 work with inline-snapshot #338

15r10nk · 2024-10-23T18:08:29Z

this is currently all work in progress ...

it is important that the actual inline-snapshot package is NOT part of the venv.

steps to create the venv:

... inside inline-snapshot repo using the mutmut3 branch

python -m venv mutmut_venv
source mutmut_venv/bin/activate.sh
python -m pip install ../mutmut # my local mutmut fork
mutmut run # which will fail but copy the code
python -m pip install -r requirements.txt
mutmut run # which should work now

boxed · 2024-10-23T18:36:57Z

mutmut/__main__.py

@@ -683,6 +687,8 @@ def new_tests(self):
 class PytestRunner(TestRunner):
    def execute_pytest(self, params, **kwargs):
        import pytest
+        params+=["--rootdir=.","--inline-snapshot=disable"]


Hmm. This seems a bit weird. Do we need some config option in mutmut for this?

maybe, I think it should not be required and it could also be a problem in inline-snapshot. But this is currently useful to make things work

The --rootdir was important. I had several problems to convince mutmut to test the code under mutants/

boxed · 2024-10-23T18:38:02Z

mutmut/__main__.py

-            output_catcher.dump_output()
-            print(f'failed to collect stats. runner returned {collect_stats_exit_code}')
-            exit(1)
+    collect_stats_exit_code = runner.run_stats(tests=tests)


You can use outputcatcher.stop() to stop it from swallowing output. This is a better method for debugging I find.

Although, the real solution is to add a flag or config to turn it off globally I guess...

Thank you for the Tip

15r10nk · 2024-10-23T18:54:42Z

@boxed this is where I'm currently:

❯ mutmut run
generating mutants
    done in 9ms
⠼ collecting stats
..s..,,.,,,,.,,,,.,,,,..,,.,,,.,,.,,.,,..,,,,..,,,,..s,,.,,.,,.s.,,,,....,,,..,,,,.,,,,..,,..,,,,..s,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,...,,.,,.s.,,.....s,,.,,,.,,,,.,,.,,..,,,,.,,,,.,,,,.,,,,.,,,,.,,,..s.,,, [ 18%]
,.,,,,.,,,,..,,,,.....s,,..,,.,,.s,,.s.,,,..,,,,.,,,,.,,,,....,,.,,.,,...,,,,.,,,,.,,,,.,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,.,,.,,.,,....s.,,,,.,,,,.,,,,.,,,.,,...,,.,,..,,,,...,,,,.., [ 36%]
,,,.,,,,.,,,,.,,,,.,,.,,...s.s,,..,,,,......,,,,.,,,,..s..,,.,,.,,.,,.,,.,,,.,,,.......,,,,..,,,..,,,,.,,.,,.,,,.,,,,.,,,,..,,,.,,,.,,,.,,,,.,,,,...,,,,.,,,,.,,,,.,,,,...,,.,,.,,.,,.,,.s,,.s,,,,.s,,.s,,, [ 55%]
,.......,,,,.s,,,,.,,.,,.,,.,,.,,.,,.,,.s,,,,.,,.,,...,,.,,,,.,,,,.,,,,.,,,,.,,,,....,,...,,..s.,,,,..,,.,,.,,..,,.....s.,,,,..,,,,.,,,,...,,,,...,,,...s,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,, [ 75%]
,,.s.,,,...s,,.,,,,.,,,,.s,,,,.s..,,,,.,,,,...,,,,.,,,,.,,.....,,.,,.,,.,,.,,...s.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,,.,,,..,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,.,,.,,,,...,,.,,,,.s.,,.,,.,,,,,,,,,,,,,,,,,,,,,,,, [ 89%]
,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,.,,.,,...,,,..,,,.,,,.,,.,,.,,.,,., [ 93%]
,,.,,,..........,,,,.,,,,...,,,,.,,,,.,,,,.,,,,.,,,,.,,...,,.,,...                                                                                                                                          [100%]
352 passed, 29 skipped, 903 subtests passed in 69.02s (0:01:09)
    done
⠸ running clean tests
    done

    done
Running mutation testing
⠴ 1821/1821  🎉 0 🫥 1821  ⏰ 0  🤔 0  🙁 0  🔇 0
0.00 mutations/second

I have no Idea why it does not test the mutations.

15r10nk · 2024-10-23T20:52:12Z

I fixed some src-layout related issues (I dont know if I fixed them in the right way, but It works for me for now).
I can test the mutations now but one mutation starts to use 100% of my system memory and mutmut run stops there and is not able to test all mutations.

... something for another day.

@boxed maybe you can use the fixes here as a base for some real fixes. I don't understand your whole code base and don't know where these things should be fixed.

boxed · 2024-10-24T04:57:21Z

Cool! Yea that's great!

Timeout for mutants that introduce infinite loops isn't implemented yet. I just disabled break->continue mutation and similar to avoid the scenario as far as possible without timeout support. But I suspected that this won't work 100% and unfortunately this happened sooner than I expected 🤣

15r10nk · 2024-10-24T06:45:46Z

nice, I got the memory limit working.
Testing the mutations got also much faster, which is very cool. I always wanted to run mutmut in ci and add a "mutant kill count" badge to my readme, I think this is definitely possible now 😄.

The last open problem is propably my own pytest plugin which comes with inline-snapshot.
I have the feeling that it interacts somehow with the test runs inside mutmut. I will have a lock at it later.

boxed · 2024-10-24T07:10:58Z

Memory limit is an interesting idea. You could set it at like 20% over the current memory, as the forked process should have loaded 100% of what is needed from the parent clean test run. That could probably ship in mutmut as is.

15r10nk · 2024-10-24T07:18:07Z

The limit is per process. I experienced that the memory usage got much higher than the limit because multiple tests where using a lot of memory at the same time.
The limit should not be bigger than TotalMemory/number of parallel tests.

You could set it at like 20% over the current memory, as the forked process should have loaded 100% of what is needed from the parent clean test run.

But there is some memory which gets freed, or? and what is if the application has only one test. 20% memory would not be enough in this case

boxed · 2024-10-24T07:29:21Z

Yea you're right. It should be x% over of the peak memory usage during the clean run, which is probably hard to capture correctly.

I'm used to working with systems where the import graph dominates the memory usage, but you're right in that this is not generally true.

15r10nk · 2024-10-24T17:13:06Z

this is a bit off topic but you might like https://github.com/15r10nk/lazy-imports-lite if you have a lot of imports 😃

boxed · 2024-10-24T17:36:30Z

Ooh. That is some dark magic 🤣 Cool!

15r10nk · 2024-10-24T20:24:28Z

Do you have any idea how you could limit the time which is used to test a mutant? I think my tests getting slower because some of the parallel mutant tests are running forever.

I was also never able to finish the mutant tests.

15r10nk · 2024-10-24T20:42:42Z

I get so far and the progress stops then.

⠸ 1817/1829 🎉 1498 🫥 209 ⏰ 0 🤔 0 🙁 110 🔇 0

boxed · 2024-10-25T05:31:50Z

I think the idea forward is to implement something similar to how mutmut2 worked: time the full test run, and forcibly kill all workers that have run longer than the full time * 1.5 or something like that. There was some rule to have a minimum time added before the timeout value was implemented to handle really small test suites, but that was mostly it I think.

15r10nk · 2024-10-25T06:28:41Z

but the problem is that the whole thing can block if you limit the number of parallel processes to 15 maybe and 15 process decide to never terminate. Mutmut will not be able to handle the remaining mutations any more.

What about this:

you know which tests you are running and I think you also know how long they run normally.
you can take this time (times 5 maybe) and use this as a limit for the process runtime.

boxed · 2024-10-25T06:30:57Z

but the problem is that the whole thing can block if you limit the number of parallel processes to 15 maybe and 15 process decide to never terminate

That's what I was talking about yea. The master process that forks off children will never run mutation. It should store start time for each PID, and if a child has lived longer than the timeout it sends kill -9 to it and marks it as timed out in the result.

15r10nk · 2024-10-25T06:41:18Z

I have fixed some src-layout related issues in my branch but I have still problems with mutmut browse probably caused by this changes.

(I added the traceback there)

I have the assumption that this caused by some general bug how some keys are generated or so.
Could you take a look at it If you have time?

boxed · 2024-10-25T07:40:57Z

Ah I think you need to delete the mutants folder and try again. The name mangling was changed and you have the old stuff on disk.

15r10nk · 2024-10-25T09:53:15Z

I delete it everytime before I run mutmut run. I tried to fix some src. related issues in this branch but I thing I fixed the problem always in the wrong way.

boxed · 2024-10-25T10:02:22Z

I pushed a few of fixes on main.

boxed · 2024-10-25T18:06:04Z

resource.RLIMIT_CPU looks very interesting as a way to implement timeout totally in the worker process.

15r10nk · 2024-10-25T18:15:18Z

mutmut browse works now also for me. I added also some diff syntax highlight to the browser.

boxed · 2024-10-25T19:04:55Z

Nice. I looked into syntax highlighting, but I failed at finding it heh

boxed · 2024-10-25T19:24:07Z

It's weirdly recollecting stats every single time. So something is wrong there.

In any case, I got it to run through the entire test suite (if I rm mutants/ before)!

⠦ 1829/1829  🎉 1473 🫥 209  ⏰ 0  🤔 0  🙁 147  🔇 0

all changes for that are on main.

15r10nk · 2024-10-25T21:03:00Z

I squashed everything into 2 commits. The fixes and the diff-syntax support.
Let me know what you think.

15r10nk · 2024-10-25T21:11:10Z

damn, mutmut is not deterministic (at least not for inline-snapshot)

I run rm mutants -r ; mutmut run twice and got the following results

⠋ 1829/1829 🎉 1355 🫥 209 ⏰ 0 🤔 0 🙁 265 🔇 0
⠋ 1829/1829 🎉 1322 🫥 209 ⏰ 0 🤔 0 🙁 298 🔇 0

boxed · 2024-10-26T01:46:51Z

Well that's interesting to say the least!

boxed · 2024-10-26T02:33:04Z

I get this a lot:

  File "/Users/boxed/Projects/mutmut/mutmut/__main__.py", line 138, in limit_memory
   resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard))
ValueError: current limit exceeds maximum limit

Maybe the memory limit idea isn't good? The resource CPU time limit should cover this problem better no? Because if the process starts allocating a lot and that starts to swap and stuff it will hit the CPU limit pretty fast anyway. Or am I not understanding the issue?

boxed · 2024-10-26T02:37:29Z

I cherry picked the syntax highlighting fix into main.

boxed · 2024-10-26T02:53:16Z

I released 3.2.0. This version runs through inline-snapshot/mutmut3:

⠹ 1829/1829 🎉 1461 🫥 209 ⏰ 0 🤔 0 🙁 159 🔇 0

It does weirdly print a bunch of output for some failed mutants, which I don't understand.

15r10nk · 2024-10-26T05:32:46Z

The reason for the output is that inline-snapshot comes with a pytest plugin which gets also mutated during test.
What you see are exceptions during the pytest run. They are not catched like normal test failures because they happen outside the tests.

15r10nk · 2024-10-26T07:26:50Z

Some of my fixes are still necessary for inline-snapshot.

This is what I get with the current main:

  File "/home/frank/projects/inline-snapshot/mutants/src/inline_snapshot/__init__.py", line 46, in <module>
    from ._code_repr import customize_repr
  File "/home/frank/projects/inline-snapshot/mutants/src/inline_snapshot/_code_repr.py", line 562, in <module>
    def _(value: Enum):
  File "/home/frank/projects/inline-snapshot/mutants/src/inline_snapshot/_code_repr.py", line 410, in customize_repr
    result = _mutmut_trampoline(x_customize_repr__mutmut_orig, x_customize_repr__mutmut_mutants, *args, **kwargs)
  File "/home/frank/projects/inline-snapshot/mutants/src/inline_snapshot/_code_repr.py", line 12, in _mutmut_trampoli
ne
    record_trampoline_hit(orig.__module__ + '.' + orig.__name__)
  File "/home/frank/projects/mutmut/mutmut/__main__.py", line 136, in record_trampoline_hit
    if mutmut.config.max_stack_depth != -1:
AttributeError: 'NoneType' object has no attribute 'max_stack_depth'

inline-snapshot runs some tests in a subprocess where mutmut is not initialized.

boxed · 2024-10-26T07:56:23Z

Ah. Crap. The reason for that is that I have that setting in my conf locally. My bad.

boxed · 2024-10-26T10:07:02Z

No, that's not it. I misremembered. Then I don't understand.

boxed · 2024-10-26T10:11:23Z

inline-snapshot runs some tests in a subprocess where mutmut is not initialized

Well this crash may be a hitbug then. If we make config load automatically, or if we ignore it, then we will have a silent error in the trampoline hit code, so mutmut won't get a trampoline hit registered. That's worse.

I don't know what the solution to that is. I don't want cross-process communication for this if it can be avoided.

15r10nk · 2024-10-26T10:33:20Z

The mutliprocess support of coverage.py uses different coverage files per process.

Could mutmut use a seperate file mutmut-stats-$pid.json for each process and combine these files after the stats run is finished?

The location of these files should be derived from an env variable because the process can have a different working directory.

boxed · 2024-10-26T14:16:02Z

I would be worried about pid based filenames about reuse. Maybe an env variable could give a full path to the file and the master process decides the name to avoid collisions.

boxed · 2024-10-26T14:19:47Z

mutmut/__main__.py

@@ -957,36 +957,59 @@ def should_ignore_for_mutation(self, path):
        return False


-@lru_cache()
-def read_config():
+def config_reader():


Doesn't this mean it ONLY reads from pyproject.toml and now ignores setup.cfg? That's no good imo. If mutmut should force pyproject.toml it should at the very least error out if there is a mutmut section in setup.cfg.

no, the config reader is your old s() function. It returns the old setup.cfg reader if there is no pyproject.toml with an [tool.mutmut] key.

Cool. I cherry-picked this fix into main

you can also cherry-pick my fix for the debug option.

15r10nk · 2024-10-26T16:13:48Z

I would be worried about pid based filenames about reuse

There can never be two subprocesses with the same pid at the same time.
But you are right that a process should probably read an existing file and add new data to it instead of overwriting an existing one.

boxed · 2024-10-26T16:30:22Z

Ah. True. If it just reads before writing then pid reuse would be totally fine. Maybe even better.

boxed · 2024-10-26T17:40:02Z

I added a debug config option to mutmut on main that will turn off all output swallowing, and passes -vv to pytest.

boxed reviewed Oct 23, 2024

View reviewed changes

15r10nk force-pushed the inline-snapshot-fixes branch from 4ef49cf to a813871 Compare October 23, 2024 18:52

15r10nk force-pushed the inline-snapshot-fixes branch from 218fc18 to d9a5ba8 Compare October 25, 2024 17:19

15r10nk force-pushed the inline-snapshot-fixes branch from d9a5ba8 to bc308a1 Compare October 25, 2024 20:43

15r10nk marked this pull request as ready for review October 25, 2024 21:00

15r10nk force-pushed the inline-snapshot-fixes branch from bc308a1 to 65737b2 Compare October 26, 2024 07:26

boxed reviewed Oct 26, 2024

View reviewed changes

15r10nk added 2 commits October 26, 2024 19:57

fixes for src-layout support

2014b74

test: fixed some tests and added hammett to the test_requirements

b94fbc4

15r10nk force-pushed the inline-snapshot-fixes branch from 09065b9 to be0e4e6 Compare October 26, 2024 18:01

fix: read booleans from setup.cfg and pyproject.toml

de58b19

15r10nk force-pushed the inline-snapshot-fixes branch from be0e4e6 to de58b19 Compare October 26, 2024 18:03

fixes to make mutmut3 work with inline-snapshot #338

Are you sure you want to change the base?

fixes to make mutmut3 work with inline-snapshot #338

Conversation

15r10nk commented Oct 23, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

15r10nk commented Oct 23, 2024

15r10nk commented Oct 23, 2024

boxed commented Oct 24, 2024

15r10nk commented Oct 24, 2024

boxed commented Oct 24, 2024

15r10nk commented Oct 24, 2024

boxed commented Oct 24, 2024

15r10nk commented Oct 24, 2024

boxed commented Oct 24, 2024

15r10nk commented Oct 24, 2024

15r10nk commented Oct 24, 2024

boxed commented Oct 25, 2024

15r10nk commented Oct 25, 2024

boxed commented Oct 25, 2024

15r10nk commented Oct 25, 2024

boxed commented Oct 25, 2024

15r10nk commented Oct 25, 2024

boxed commented Oct 25, 2024

boxed commented Oct 25, 2024

15r10nk commented Oct 25, 2024

boxed commented Oct 25, 2024

boxed commented Oct 25, 2024

15r10nk commented Oct 25, 2024

15r10nk commented Oct 25, 2024

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

15r10nk commented Oct 26, 2024

15r10nk commented Oct 26, 2024 • edited Loading

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

15r10nk commented Oct 26, 2024

boxed commented Oct 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

15r10nk commented Oct 26, 2024

boxed commented Oct 26, 2024

boxed commented Oct 26, 2024

15r10nk commented Oct 23, 2024 •

edited

Loading

15r10nk commented Oct 26, 2024 •

edited

Loading